Baguamarsh: An Immersive Narrative

Visualization for Conveying Subjective

Experience

Fei Jiang

1,2(&)

, Don Derek Haddad

, and Joseph Paradiso

Shanghai Academy of Fine Arts, Shanghai, China

fred_jf@hotmail.com

MIT Media Lab, Cambridge, USA

Abstract. As ubiquitous sensing becomes embedded in our everyday world,

we can easily obtain multimodal sensor data from our physical surroundings.

Objectivity is a core value in the visualization research community. However,

personal experiences are characterized not only by objective facts, but by per-

sonal emotions. In this paper, we explore immersive data visualization for

conveying personal subjective perception and experience by using multidi-

mensional data and multimedia. We introduce a framework to describe narrative

structures in immersive data visualization, and provide an example project

Baguamarsh as a proof of theme-based creative methods for designing corre-

lations between different information. We hope this study may offer outline

opportunities for future research in narrative visualization.

Keywords: Virtual reality

 Narrative visualization  Multidimentional data 

Storytelling  Bagua  Book of Changes

1 Introduction

As ubiquitous sensing becomes embedded in our everyda y world, we can easily obtain

multimodal sensor data from our physical world, as personal health data, meteoro-

logical data, geographical data and etc. There are many approaches that have been

designed to present date for users to communicate information clearly and efﬁciently.

Data visualizations are commonly presented quantitatively, as statistical graphi cs, plots,

charts and other tools [1]. Objectivity is a core value in the visualization research

community [2]. However, personal experiences are characterized not only by objective

facts, but by personal emotions [3]. How to convey personal subjective perception and

experience by using multidimensional data becomes both an opportunity and a chal-

lenge in visualization ﬁeld.

On the one hand, storytelling is an effective way of conveying information and

enhancing understanding [4], as narrators can leverage personal information (data,

photos, audio, etc.) and express their subjective ideas through narrative visualization.

On the other hand, as VR headset s have become affordable for individuals, virtual

reality becomes a user-friendly platform for reconstructing and representing digital

information.

M. Kurosu (Ed.): HCII 2020, LNCS 12181, pp. 596–613, 2020.

https://doi.org/10.1007/978-3-030-49059-1_44

In this paper, our approach treats the data visualization as a narrative and uses

virtual reality as a platform to convey a narrator’s story. Firstly, we discuss related

work on visualization research. Secondly, we introduce a framework to describe nar-

rative structures in immersive data visualization. Finally, we provide an example

project “Baguamarsh” as a proof of concept. Each of these aspects plays a great role in

allowing the narrator to affect the observer emotionally and intellectually by conveying

a rich, signiﬁcant idea.

2 Related Work

2.1 Pre-research

Project “Moments” is a prototype system for saving and representing personal

moments [5] and presents an overview of a whole process of creating an immersive

data visualization, whi ch provide a good reference value for this study.

A wireless sensor network (deployed at Tidmarsh Wildlife Sanctuary in south-

eastern Massachusetts) [6], a nd Chain-API (a RESTful service providing sensor data) ,

provide a place instrumented with a dense senor network for ﬁeld study and an open-

data platform for collecting environmen tal data, hence provides an ideal setting for this

project.

2.2 Subjectivity in Narrative Visualization

As narrative visualizations combine patterns of communicative and exploratory

information visualization to convey intended stories [4], many communities have

commented on the importance of narrative in data visualization [9]. But according to

Tong et al. [7], comparing with the development in other ﬁelds, storytelling is a

relatively new subject in visualization. On one hand, even storytelling has become a

common topic of discussion in data visualization now, but most research commonly

represents objective facts [7], and there is only a little research on personal storytelling

[3]. On the other hand, there are some data artworks that express subjectivity as a

design goal, but they do not describe speciﬁc techniques and methodologies for cre-

ating subjective representation [3].

2.3 Immersive Data Visualization

Virtual reality not only has been shown to lead to better discovery in domains whose

primary dimensions are spatial [8], but also to enhance situation awareness [9] and

media richness [10]. On one hand, according to Donalek et al., as virtual reality can

maximize the intrinsic human pattern recognition skills [8], immersive data visual-

ization provides more intuitive data understanding than traditional “desktop” visual-

ization tools. On the other hand, the public has understood that virtual reality already

portended a new medium for almost two decades [11], and we have seen much research

Baguamarsh: An Immersive Narrative Visualization 597

on interactive experience, presence, immersion and interaction [12–17], but not many

works that address immersive narrative visualization involve live sensor data.

3 Framework

We explore the use of immersive virtual reality platforms for narrative visualization to

convey subjective experience. To achieve our goal, we introduce a theme-based

framework (shown in Fig. 1) with which a user not only can convey personal sub-

jective perception and experience by using multimedia and multidimensional data, but

also can enhance multidimensional expression.

According to the commonly used pipeline for collecting data [3] and creating

information visualizations [18], we inject a theme-based method in workﬂow to rep-

resent subjective personal narratives in an immersive environment. Firstly, a main

theme chosen by a narrator is the selection criteria for collecting information (data,

media, etc.). Secondly, according to the theme, the narrator can choose to simplify,

classify, con nect or reconstruct information materials. Thirdly, rhetorical strategy is

used to map all content for representing annotation, user interface, presence and

interactivity. Finally, perceptualization is integrated for building up an immersive and

interactive environment.

4 Project Introduction

In this section, we present a project “Baguamarsh” that exempliﬁes the goals described

in last section. This project takes the idea of the unity between heaven and man of the

ancient Chinese philosophy I Ching (Book of Changes) [19], combines the Bagua

(Eight Trigrams) of the I Ching with multidimensional data and multimedia, and uses

an immersive interactive environment to present a novel form of narrative visualization.

The main theme of this project is the “Unity between Heaven and Man”, namely the

harmony of nature and human. It systematically expounds the relationship between

man and nature, that man must follow the laws of the universe, to respect and protect

nature, and to have an insight into the truth that harmony can produce all things.

According to the main theme, we designed this project under the form of a virtual

reality environment by integrating personal data, environmental data and media

materials.

An I Ching framework has been used in prior work to interpret and represent

wearable sensor data for a quantiﬁed self application [20], but not to our knowledge for

integrated personal and large-scale environments sensing.

598 F. Jiang et al.

5 Information Collection and Processing

The data used for this project is divided into user data and environmental data. The user

data is mainly from wearable devices, that is to say an Apple Watch 4 and a Muse

BioHarness in this project. These two kinds of devices are separately responsible for

collecting a narrator’s ECG and EEG. Environmental data is mainly obtained in two

ways: (1) a GoPro 360 camera is responsible for taking 360° panoramic photos and

recording ambisonic audio from user environmen ts (shown in Fig. 2), (2) Using the

open data interface to obtain environmental data through time and GPS information

encoded in the panoramic photo EXIF information.

Fig. 1. Theme-based framework for constructing immersive narrative visualization

Baguamarsh: An Immersive Narrative Visualization 599

After the narrator wearing the wearable devices enters the data collection area

(Tidmarsh), he/she ﬁrstly takes a panoram ic photo with the 360 camera, and then starts

recording a panoramic video of about 30 s in the same position (the video contains

ambisonic audio) (show n in Fig. 3). Dur ing the video recording process, he/she needs

to collect ECG and EEG data for about 30 s when the mobile phone is connected with

the wearable device. The narrator may take advantage of the features of the 360°

camera, as he/she doesn’t need to focus or frame when shooting image material, so

he/she can collect other data at the same time, which greatly shortens the information

collection tim e. After actual operation, we ﬁnd that the information collection process

for one location usually takes no more than 2 min.

Fig. 2. Devices for information collecting

Fig. 3. Information collection in Tidmarsh

600 F. Jiang et al.

Information processing is mainly divided into three steps: information extraction,

data expansion and data archiving (shown in Fig. 4). (1) Information extraction: mainly

for the image materials captured by the 360° camera, which includes panoramic photos

and panoramic videos. The panorama photo is a JPG image ﬁle, and each ﬁle contains

EXIF information. We use a program written in Python to extract the time and GPS

information from each photo for later use in data expansion. The extraction process for

panoramic video ﬁle information is relatively time consuming. We import the

panoramic video into the post-processing software Premiere before extracting the 4-

channel ambisonic audio. In addition, we use a sound generation program written in

Python to convert the ECG signals collected in the previous period (Fig. 4, top) into 30-

s heartbeat sounds, which will be used in the later virtual reality interacti on. (2) Data

expansion: using the time and GPS information in the previous photos, we obtain the

24-h environmental information of the information collection day through an open data

interface. The open data platforms used in this project include the following: Google

Maps API, AccuWeather APIs, Chain API [21]. We use Python-written programs to

get locations and elevations from the Google Map API, get temperatures and winds

across the entire area from AccuWeather APIs, and get the pressure, relative humidity,

visible light, intensity of infrared and ultr aviolet light, soil temperature and humidity

from the Chain API in the area where the user is located at Tidmarsh. We put all of this

data together to generate a text ﬁle in JSON format. (3) Data archiving: We use the

desktop program written by Unity to package panoramic photos, ambisonic audio and

text ﬁles containing all metadata into an archive ﬁle, which is convenient for subse-

quent virtual reality program calls.

6 Content Mapping

The way we present content has the following characteristics: (1) Narrative visual-

ization will be presented in the VR environment reconstructed from 360 panoramic

photos, and users can switch between the real environment and special effects envi-

ronment. (2) By combining the physical properties of the phenomena described by the

data with commonly used visualization methods, we develop novel and user-friendly

modes of presentation. (3) Eight Trigrams (see below) will persist across the entire

visual representation, and the symbols corresponding to each kind of trigram will be

adopted as the basic visual elements of the environmental data points, which also

reﬂect the connection between data and trigrams on the micro level and make the

observation more convenient, thereby improving annotation. (4) Most of the visual-

ization objects in this project are time series datasets in the unit of scenes. All data

points can not only display the visualization effects at different times and in different

trigrams, but also reﬂect the relationship between personal data and environmental data,

thereby abstracting a visua l pattern of “Unity between Heaven and Man” to users.

Baguamarsh: An Immersive Narrative Visualization 601

Fig. 4. Information processing

602 F. Jiang et al.

Eight Trigrams are the basic concept of the Book of Changes and can represent the

dynamic and still states of all natural phenomena. Eight kinds of symbols used in the

Eight Trigram correspond to eight phenomena in nature (Heaven, Earth, Water, Fire,

Thunder, Wind, Mountain, Marsh), and their names are “Qian, Kun, Kan, Li, Zhen,

Xun, Gen and Dui”. Through the evolution of these eight natural phenomena, we

speculate on the laws of man and nature in an attempt to achieve a world view of

harmony between humanity and nature. The Eight Trigrams are divided into Earlier

Eight Trigrams and Later Eight Trigrams. What is used in the project is the Later Eight

Trigrams, which are said to have been made by Zhou Wenwang. In the Later Eight

Trigrams, the Trigram Zhen is the starting point and it is due east. According to the

clockwise direction order, there are in sequence: Trigram Xun–Southeast; Trigram Li

—South; Trigram Kun–Southwest; Trigram Dui—West; Trigram Qian–Northwest;

Trigram Kan—North; Trigram Gen–Northeast. The order of the Later Eight Trigrams

are: one of Kan, two of Kun, three of Zhen, four of Xun, ﬁve of the center, six of Qian,

seven of Dui, eight of Gen, and nine of Li. The ancients used to draw maps with south

at the top and east in the left. We create a three-dimensional interactive menu based on

graphics of Eight Trigrams and the shape of the Fengshui Bagua mirror for the

interaction of the VR controller (shown in Fig. 5).

According to the eight phenomena descri bed in the Bagua, we divide the acquired

environmental data into eight categories that correspond to each of the Trigrams

(shown in Fig. 14) in the menu: (1) Trigram Qi an: atmospheric pressure (2) Trigram

Dui: soil moisture (3) Trigram Li: UV index (4) Trigram Zhen: cloud cover (5) Trigram

Xun: wind speed (6) Trigram Kan: humidity (7) Trigram Gen: elevation (8) Trigram

Kun: soil temperature. Users can interact with the data space through the Bagua menu:

the speciﬁc visualizations are as follows (Figs. 6, 7, 8, 9, 10, 11, 12 and 13):

Fig. 5. Bagua menu

Baguamarsh: An Immersive Narrative Visualization 603

Fig. 6. Trigram Qian – atmospheric pressure (AP): the atmospheric pressure data is used to

control the volume of a ring around an object. The higher the pressure, the larger its volume.

Fig. 7. Trigram Dui – soil moisture (SM): the soil moisture data is used to simulate the water on

the ground. The greater the moisture value, the wider the line of water.

Fig. 8. Trigram Li – UV index (UV): the UV index represents twinkling lines in the sky. The

higher the value, the high their density.

Fig. 9. Trigram Zhen – cloud cover (CC): the value of cloud cover is used to control the density

of clouds in the sky. Cloud density increases with the value of cloud cover.

604 F. Jiang et al.

Fig. 13. Trigram Kun – soil temperature (ST): the soil temperature is used to control the color

tone of ground material. The higher the temperature, the warmer the color.

Fig. 10. Trigram Xun – wind speed (WS): wind data is used to control the rotational speed of

visual elements. The higher its value, the faster the rotational speed.

Fig. 11. Trigram Kan – humidity (Hu): humidity is used to control the density of visual

elements. The larger the value, the higher the density.

Fig. 12. Trigram Gen – elevation (El): elevation data controls height differences between water

particles and other objects. The higher the value, the bigger the gap.

Baguamarsh: An Immersive Narrative Visualization 605

According to the I Ching, Trigram Qian corresponds to the body’s brain, and

Trigram Li corresponds to the heart. We build the EEG and ECG into a dynamic

model, then place them at the cen tre of the interactive menu. Both the EEG and ECG

data are displayed as line graphs around the model, and all environment data is shown

on the submenus attached to the interactive menu (shown in Fig. 5).

The green main menu can be rotated by the VR controller. Each Trigram contains

different kinds of data and different combinations of the Five Elements which are

designed for triggering visual effects.

Users can select the environmental data within 24 h of the day of the encounter

through the menu selection. We map the elements represented by each trigram to the

relationship between the ﬁve elements. We superimpose the elements corresponding to

these attributes over the visual elements manifested by different data to re ﬂect the

process of mutual promotion and restraint between the ﬁve elements (shown in Fig. 15

and Fig. 16), reﬂected by the Eight Trigrams. The calculation formulas for each Tri-

gram are as follows:

Qian: fAPðÞ¼AP þ EEG þ El  UV ð1Þ

Dui: fSMðÞ¼SM þ ST  UV ð2Þ

Li: fUVðÞ¼UV þ ECG þ CC þ WS  Hu ð3Þ

Zhen: fCCðÞ¼CC þ Hu  SM ð4Þ

Xun: fWSðÞ¼WS þ Hu  AP ð5Þ

Kan: fHuðÞ¼Hu þ AP þ SM  ST  El ð6

Gen: fElðÞ¼El þ UV  CC ð7Þ

Kun: fSTðÞ¼ST þ UV  WS ð8Þ

606 F. Jiang et al.

Fig. 14. Data mapping of eight Trigrams

Fig. 15. The production and destruction relations in ﬁve elements

Baguamarsh: An Immersive Narrative Visualization 607

Before we use the algorithm, there are very little differences between each time

point, so users can hardly feel the interaction in VR. By using the calculation formulas

of the mutual promotion and restraint between different data, the visual effect changes

dramatically, which can provide the user a better interactive experience. When a user

chooses a different Trigram, the lines between each element will be changed. The green

line represents production and the red line represents destruction (shown in Fig. 16).

Fig. 16. The algorithm of relations for data interaction

608 F. Jiang et al.

7 Immersive Interaction

For highly immersive visua lization purposes, a ﬁrst-person view is appropriate (shown

in Fig. 18). The observer will experience both visual and audio cues. This project is

currently available as a prototype application for Oculus Go, which has a relatively

high performance/price ratio on the market. This VR headset does not need to be

connected to a computer, which makes a better user experience. In terms of software

production, we use Unity3D to import the archive ﬁles that were previously packaged

by the desktop program into the VR main program in batches, and generate an Android

program that supports Oculus Go. Based on the characteristics of the device, we design

a VR interactive mode according to the mode chosen in the menu.

When a user is in the VR environment, he/she can not only walk around, look

around, hear 3D background sound, selec t scenes, and hide or move visual objects, but

also can select different trigrams to explore different visualization and auralization

results. For example, when a user chooses Trigram Li, it will begin to rain Trigrams

(shown in Fig. 19). He/she can hear the sound of heartbeats and a UV Index data value

is loaded for the interactive menu. The users can switch between the real environment

and the data environment (shown in Fig. 21), or change the time to see what would

happen to the environment then. He/she can even use the controller cursor to select a

3D object to change its appearance (shown in Fig. 20) based on the value of Trigram

that he/she chose.

Fig. 17. Different options presents different hints (Color ﬁgure online)

Baguamarsh: An Immersive Narrative Visualization 609

Fig. 18. First-person view

Fig. 19. Trigram rain for notiﬁcation

610 F. Jiang et al.

Fig. 21. Sky of data environment

Fig. 20. Change the appearance of a tree

Baguamarsh: An Immersive Narrative Visualization 611

8 Conclusion

In this paper, we introduced a framework and an ongoing project to describe how to

design an immersive narrative visualization for conveying subjective experience. We

describe the whole process of project Baguamarsh for deliberately expressing subjec-

tivity during information collection, processing, interaction and presentation. These

preliminary studies show us the ﬁrst insights into the potential of immersive data

visualization as a subjective storytelling platform and prompt a discussion for future

research on conveying subjective experience in personal visual storytelling. More

visuals and video from this project can be view ed at https://www.media.mit.ed u/

projects/baguamarsh/overview/.

Acknowledgments. This study is supported by The Responsive Environments Group at

The MIT Media Lab. This group mainly explores how sensor networks augment and mediate

human experience, interaction, and perception, while developing new sensing modalities and

enabling technologies that create new forms of interactive experience and expression.

References

1. Friendly, M.: Milestones in the history of thematic cartography, statistical graphics, and data

visualization (2008)

2. Jorgenson, L., Kritz, R., Mones-Hattal, B., Rogowitz, B., Fraccia, D.F.: Is visualization

struggling under the myth of objectivity? In: Proceedings of IEEE Visualization (1995)

3. Thudt, A., Perin, C., Willett, W., Carpendale, S.: Subjectivity in personal storytelling with

visualization. Inf. Design J. 23(1), 48–64 (2017)

4. Segel, E., Heer, J.: Narrative visualization: telling stories with data. IEEE TVCG 16(6),

1139–1148 (2010)

5. Jiang, F.: A prototype system for saving and representing personal moments. In: Marcus, A.,

Wang, W. (eds.) HCII 2019. LNCS, vol. 11585, pp. 314–322. Springer, Cham (2019).

https://doi.org/10.1007/978-3-030-23538-3_24

6. Mayton, B., et al.: The networked sensory landscape: capturing and experiencing ecological

change across scales. Presence Teleoper. Virtual Environ. 26(2), 182–209 (2017)

7. Tong, C., et al.: Storytelling and visualization: an extended survey. Information 9, 65 (2018)

8. Donalek, C., et al.: Immersive and collaborative data visualization using virtual reality

platforms. In: 2014 IEEE International Conference on Big Data (2014)

9. Endsley, M.R.: Toward a theory of situation awareness in dynamic systems. Hum. Factors:

J. Hum. Factors Ergon. Soc. 37(1), 32–64 (1995)

10. Klein, G., Militello, L.: Some guidelines for conducting a cognitive task analysis. Adv. Hum.

Perform. Cognit. Eng. Res. 1, 161–199 (2001)

11. Bates, J.: Virtual reality, art, and entertainment. J. Teleoper. Virtual Environ. 1(1), 133–138

(1991)

12. Jennett, C., et al.: Measuring and deﬁning the experience of immersion in games. Int.

J. Hum.-Comput. Stud. 66, 641–661 (2008)

13. Usoh, M., Catena, E., Arman, S., Slater, M.: Using presence questionnaires in reality.

Teleoper. Virtual Environ. 9(5), 497–503 (2000)

612 F. Jiang et al.

14. Garau, M., Slater, M., Vinayagamoorthy, V., Brogni, A., Steed, A., Sasse, M.A.: The impact

of avatar realism and eye gaze control on perceived quality of communication in a shared

immersive virtual environment. In: Proceedings of the 2003 Conference on Human Factors

in Computing Systems, CHI, Ft. Lauderdale, Florida, USA (2003)

15. Heidicker, P., Langbehn, E., Steinicke, F.: Inﬂuence of avatar appearance on presence in

social VR. In: 2017 IEEE Symposium on 3D User Interfaces (3DUI). IEEE (2017)

16. Schubert, T., Friedmann, F., Regenbrecht, H.: The experience of presence: factor analytic

insights. Teleoper. Virtual Environ. 10(3), 266–281 (2001)

17. Witmer, B.G., Singer, M.J.: Measuring presence in virtual environments: a presence

questionnaire. Teleoper. Virtual Environ. 7(3), 225–240 (1998)

18. Card, M.: Readings in Information Visualization: Using Vision to Think. Morgan

Kaufmann, Burlington (1999)

19. Wilhelm, H., Baynes, C.F., Jung, C.G.: The I Ching or Book of Changes. Princeton

University Press, Princeton (1997)

20. Yano, K.: AI for taking on the challenges of an unpredictable era. Hitachi Rev. 65,35–39

(2016)

21. Russell, S., Paradiso, J.A.: Hypermedia APIs for sensor data: a pragmatic approach to the

web of things. In: Proceedings of the 11th International Conference on Mobile and

Ubiquitous Systems: Computing, Networking and Services (2014)

Baguamarsh: An Immersive Narrative Visualization 613